DOCUMENT RESUME 

ED 070 440 JC 730 013 



AUTHOR 
TITLE 



INSTITUTION 
REPORT NO 
PUB DATE 
NOTE 



Landiniy Albert J.; Gold, Ben K. 

Proposed Evaluation Design for Los Angeles City 

College's Auto-Tutorial Programs and Learning 

Resources Center. 

Los Angeles City Coll. , Calif. 

LACC-RS-72-17 

Dec 72 

15p. 



EDRS PRICE 
DESCRIPTORS 



MF-$0.65 HC-$3.29 

♦Autoinstruc tional Programs; Behavioral Objectives; 
♦Course Evaluation; ♦Evaluation Techniques; Flow 
Charts; Formative Evaluation; Guides; ♦Instructional 
Materials Centers; Junior Colleges; Post Secondary 
Education; Research; ♦Summative Evaluation 



ABSTRACT 

A guideline to a two-stage outcome evaluation for 
individual courses within a total Learning Resources Center project 
is presented. Program and process variables are taken into 
consideration with emphasis on evaluating change in student behavior. 
The use of behavioral objectives is cited as the best means of 
determining the difference between treatment groups, and a scheme for 
using them is advanced. Two flowcharts are provided to serve as 
guides in conducting the evaluation. (Author/DB) 
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PROPOSED EVALUATION DSSIfflJ FOR LOS ANGELK CITY COLLEGE'S 
AUTO-TUTORIAL PROGRiU^S MD LEARNING RESOURCES CENTER 

INTRODUCTION c 

This report i,s the second in a aeries of reports being produced by the 
Lo3 Angeles City College (LACC) Research Office dealing with the evaluation 
of that school's Auto-Tutorial (A-T) programs and Learning Reaourcea Center 
(IRC)« Action surrounding these programs has largely been the responsibility 
of the Media-Oriented Systems Technology (MOST) Task Force convened to develop 
and ioplement media oriented instructional programs r. 

The problem ^ As the development of A-T programs for the LRC has prcgres86ci 
the MOST group began to evidence greater concern, for the evaluation of their 
efforts » The first step in the ensuing evaluation vas conduction of a needs 
assessment to determine the broad goals of the overall prcgram«* Within the 
time constraints isqiosed on the project it \v*as thought that the second formal 
step in the evaluation project should be a documentation of the anticipated 
sumaative or outcome evaluation design^ 

Statement of purpose o In this paper the evalixatlon concerns for the A-T, 
LRC project are stated and described^ In addition, proposed evaluation de- 
signs are presented for both the individual A*T programs as yell as the overall 
UtC prolect o It is anticipated that these will serve as guidelines for those 
persons actually conducting the final outcome evaluation for the MOST group » 
The designs presented here should not be considered final statements as it is 
likely that program iiqilementation evaluation and program progress evaluation 
will cause the final evaluation design to be modified « 

^Landlnly Albert J. and Ben K. Gold, An Ex Post gacto Needs Assessment 
Psing a Modified Delphi Technique to Determine the Goals of a Community Col - 
lege Learning Resources Center^ Los Angeles City College » Research Study 
?72-6, Kay, 1972. 
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11. PROGRAM GCALS 

GoalG developed for the over^iil LRC project vo'jce cast within the frame- 
work that the overrJ-ding principle concern5-ng the LRC, as understood by the 
research staff and approved by the MOST group, v/ais that the initial decision 
to construct an LRC at L&CC T/a-^ to be able^ tc;^ reg^^ to increasing student 
enrollments in the f ace of diiaj .nlshlnf;; reyeijiue^ without lowering the quality 
of education c?££eredo 

Global JSoal areas o In the needs asseat^meat phase of this evaluation pro- 
ject several global goal areas were identified.^ These were the areas in which 
it was anticipated that the LRC vould liave a poi*itive effect on student's skills 
attitudes and knowledge They were identified as: 

* Small student dlscustiiion groups 

* Better learning envirozuaent 

* Better instruction 

* Individualized Instruction (machinery) 

* Individualized instruction (instructor) 

* Better organizational structure 

From these 9 final goal statements representing the global goal areas were 
constructedo 

Flmtl goal statements g The final goal statements represented complete state- 
ments or£ the global goal areas and were stated thu8ly<. Los Angeles City College 
courses tising the new Learning Resources Center and Auto-*Tutorlal services will: 

* Benefit from the establishment of Increased 
student-^teaeher interaction through the forma- 
tion of small student discussion groups $ occur- 
Ing at no additional cost to the instructional 
program 

Bxperience an increase in professional quality 
' and humanlzatlon of instruction for their in- 
dividual students 

* Witness a general improvament of instructional 
quality through Improved course content, periodic 
subject matter review, and heightened creativity 
in communicating subject matter material to stu- 
dents 
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Final goal Gtatemerjts (coni:lnaed; 

Have instructors who r«..vu'uin3iy updarti ap- 
proaches and use necj teaching \\'echnique0 
in better organized courses 

* Allcw students to tcove through and reviev; 
ccurse material at their ovm pace regard-* 
les^ of their individual ability level, know- 
ing if they have completad a sectioii succesB- 
fully or need to repeat it 

* Realize increased tJtudent gain., by having Im- 
proved learning situations that taake educa - 
tion more interesting and dtlTnulating 

It was this set of statements that constituted the final product of the 

needs asse£S6i£^nt« 

Positive objectives > In ino\-lns Sirom the needs assessment to the Gut^^ome 
evaluation design it was found necessary to expand each one the final goal 
statements to a series of Positive Objectives o Each of these objectives pro- 
vided a basis for precise measurement of the attainment of each goal expressed 
ear Hero They are listed here for the readers convenience.. 

* Classes using the LRC approach will have measurably 
more student- teacher interaction than classes utiliz- 
ing the traditional lecture approccch 

^ Small discussion groups of 15 to 30 students will be 
formed as sub'^unlts of classes using the LRC approach 

* The formation of small student dlscufifsion grcuj.3 will . 
result in no additional costs to the LRC Instructional 
program when compared to parallel traditional c lasses « 

* Students Involved In the LRC approach classes will have 
better self images of themselves as students and more 
favorable attitudes toward school tlian students in 
parallel traditional classes 

* Instructors assigned to I£C classes x^ll manifest ^loore 
positive attitudes and enthusiasm toward their pro- 
fession than instructors not assigned to such programs 

^ Instructors assigned to I&C classes will have a greater 
desire to aid their students in the learning process 
than Instructors not assigned to such programs 

* lite instructional programs will have more current ma- 
terial Incorporated Into them than traditional programs 

^ IRC instructional program materials will be periodically 
reviewed and updated 

* I&C Instructional programs will reflect heightened in- 

[ structor creativity In coomunlcatlng subject material to 

I students 

i 
1 
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Positive objective:^ (conLxnurid) 

^' .ln.siructort^ fceaching LRC o^au-i^rf wTJi ust inno- 
vative teaching tchniqutr^ c> gr exciter degree 
than instructors not lnv';;;i^'^r;d wi-uh t^uch pro'graTUii 
Sytjtematic managemonr pri:u i.pi^^^ v;iXl be employed 
In IRC classes to routintily update teachbog ap 
preaches 

^' Students in IRC ciasBfeS vyjl be exposed ro new 
cla93 msterial afc a pa»^e jl their otki chootiing 

* Students in IRC cUs^ei; wili h^vu^ immediate feed 
hack as to their tiuccet,^i ^i-^ Lesrnfng V:he material 
presented 

Operational st_atmgn^- '^^^ pu>^itivo ubjetci.veci ware fmther broken down 
to a concrete t.et of measurable Operational Stat:-mcnts It is anticipated 
that this lliit of i^tatement? will be the befits for tht construction of a numb 
of test items to be used in the overall assessment of the lilC program'^ :4 ef- 
fectiveness o 

* Classes using the LRC approach vill have meacfurably 
xnore student « teacher interaction than claBbes at.iliz- 
ing the traditional lecture approach 

lo There will be a difference between the IRC 
and traditional classes in the amount of 
personal interaction reported In terms of 
student-teacher contact in the classroom 
and outside the classroom by both teachers 
and students 

2. Student?^ in the LRC classes will report 
greater personal involvement with their 
instructors than students in traditional 
classes 

3. Teachers in LRC classes will report greater 
personal involvement with their students than 
teachers in 'traditional classes 

4. Teachers in IRC classes will report more hours 
of student contact on an individual basis than 
Instructors in traditional classes 

* Small discussion groups of 15 to 30 students will be 
formed as sub-units of classes using the IRC approach 

lo IRC classes will have discussion groups smaller 
than the Instructional groups associated with 
traditional lecture classes of parallel subject 
matter 

2« IRC classes will have discussion groups no 
larger than 30 students 
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* The fviritial-Um o.t l;uui11 student dlsi u.stilon group?^ ^vxil 
reault: in uo additional cost« tiy the LRC in«truv..tionfc I 
prograw tchen compared to parallel iradltional claflf^e^ 

1. i^ien cotnputec, cosv per student ^d.11 be the 
»ame cx lesj 1:or LRC clanses ay traditional 

2 Additional bud.'jiii: allrywancCB will not be made 
beyond developmental custa fo'c LRC clashes on 
a per atudent -.oBt baaifs 

* Students involved in the LRC clasBee will have better seifc" 
images of themselves a.s students 

1, Using «el£ inventory reporting instruments, students 
in LRC classes will report stronger self images than 
those in traditional courses 

2. Students in l^C courses will have higher overall 
GPA^s tor whe seaeater they are Involved in than 
students in parallel but non LRC courses 

* Instructors assigned to LRC clasaeti will manifest more 
positive attitude^i and -enthusiasm toward their pro- 
fession than instructors not assigned to such progxains 

Instructorn assigned to LRC classes will report 
reading more material in their field more often, 
attending more professional meetings , spending 
more time in developing classroom materials than 
inatructors in traditional classes 
2. Students in LRC classes will report that their 
instructors are more enthisiastic in the presen-* 
tation of materials ; interested in student well 
being and cooperative in their manner and be- 
havior than instructors in traditional classes 

* Instuctbrs assigned to LRC classes will have a greater 
desire to aid their students in the learning process 
than Ifistructors not assigned to such programs 

lo Students will report greater rapport with in- 
structors teaching LRC courses 

2. Students will report Instructors teaching IRC 
courses are easier to reach for appointments 
outside class 

3, Instructors teaching LRC coursed will report 
greater tutorial contacts than teachers not 
so Involved 
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Operational statements (c^infcinued? ^ 

* IRC inotructional progr^jns \rlll have v^:e current material 
incorporated int«i them than traditional programs 

lo Department Chairmen will report thav LRC courses 
have the most current subject matter appropriate 
Incorporated into them 

?.o The Dean of In8trui:;tion will report that LRC 

courses have the most current subject matter ap- 
propriate incorporated into them 

3o Course evaluatlcm by peer instructor groups ^rf.11 
report that LRC courses have the mcist current 
subject matter appropriate incorpocdtetd into 
them 

* LRC instructional program materials wili be periodically 
reviewed and updated 

lo LRC Instructors vrLll produce documeaiiatioti indicate* 
ing yearly updating of the audic-tuto-rlal segment of 
their course 

2o The IRC will produce documentation Indicating that 
the updating capability of its ^qulp^iient has been 
utilized to update each instructional package yearly 

* LELC instructional prcgrams will reflect heightned instruct 
tor creativity in comsminicating subjec:t matter to students 

1« Students will report greater ease in learning IRC 
course subject matter than students in parallel 
traditional courses 

2c Students will evidence greater retentive ability of 
subject matter when tested ever time 

* Instructors teaching LRC classes will use innovative teaching 
techniques to a greater degree than Instructors not Involved 
with such programs 

lo Using a rating team methodology p LRC teachers will 

be shown to use more innovative teaching techniques 

than traditional teachers 
2o Students In IRC courses will admit: to greater e3C-> 

posure of innovative teaching techniques In LRC 

courses than in traditional ones 
3o ISC Instructors will state that they are using more 

Innovative teaching techniques than instructors 

teaching traditional courses 

Systematic management principles will be employed in LRC classes 
to routinely update teaching approaches 
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•...IC i--:^--vvr v-?.;./ •3;:.j.';'..:.o ,•• pr..r;.;;^'.r'. / .cor 

.■> . ■.?:'.r.;:'.c?.<';-'"....!.y '.'.■•'j..' "' !i!'.t.".."i:in'"' f;;:'^" ''" 

ci^.cip.r; rp.a;;c'..-ia?.(: ivi Y..;i;.: c.\;i«'j-s; -d.] h<.^ va •.v.-'r.:-^'; 
as CO the?.:..- str.lo ■l:'-'-i'-t'.;;:: p-.-uccdurrj thrcv. r- h 

otri'.'ji.i.or; 

2.> Stydanto xjlll report :.ii;.i.;.-g the uelf ■^s.'-ixic- 
option o?: tiv,; L3.C ariulpr.rm;; 

Sirudeni-.s in LRC claimn^ w.!?.! he;- in,nier.li?.'ce cecdbi^cl: as 

Xv Infcarira e.x:in'xnaZxov.» rrJJ.:. L-cAivatc accrjptable 
Gtuuent Kcarjuury oi tho yabjecfc laattGi- (80/80) 
E:camlaatJ.oK£i ioi i.cv each unit o:c r.:&dia 

pi-etfenuaMon 

leiLiiiing gains thsii studentt; eiirolled in rfaj:pVeX hut 
traditional clasiiiis 

.V. Ueins approprriate standnrdiaed esaiaiuatiom (if 
available) atcdont-j in thsi LRC couruay wHi in- 
dicate greatyr :'.tiarnir.g gains than, atctdents en- 
rolled in parallel but traditional claHses 

HI. EVALUA'flOH (lONCiniNS 

In evaluation t,,o concerns are ovideat; (1) formative evaluation, 
, or the producing of information for the product developer to infom bim 
if hio efforts are reaching the ir.ark he set for himself. It's purpose is 
to aid the product developer iu n^aklng correct changes before his progran^. 
has been put in the final stngay; (2) suiraiiative or outcome evaluation, 
wliose purpose U to provide the decision mker with infornsation to aasr^er 
such questions as, -Should ve continue the prosratn ne::t year?", and "Should 
ve extend the program into other subject matter areas." 
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P.}^:i^}^^}^. :L^Ul^h.k\i ThU^ . ti. Is prlTiiarily -.-rn. 
wilth presenting a tor the » ina t u..^ rne ^Vi? Xuati.on , In -v/i.ng 

the Po3itive Obje» rj^^e.-^ and Oper^i loeuii Sca^»'?mt'-ni.^? adviAn<^;edi «arJier .r t s 
readily noted »*.uav thf- majwr crmoc-rn.^ r?i: j:Vh;= MOST Tabk Fierce ve7.r. WM.ti 
proce.iiS ViirJ.abler. fa'^h.- r than proj^raoi vai labiet^ This c^n-jem ' .» C-rcH«:iL 
tional. in educ^*?:: '-iV'a.uiaLlun bfj' n^t au?ays « c<TXii.ci one. 

'.Cn procefc;i.> »^vaiuai::.von., tht: i.-air^ernc^d educacor:^ and deo'r^xrrn niaker^^ 
are seeking int ovtaati jn regarding uhe. m^:: r i.^ u\:i»m' i* ability ixv jivt^r and 
maintain the pr^/grain This di.lifjv-^ rroni p) . gram walviition in which the 
question being a>*kf.'d //mply /.ht» pi*.;grain better .. \oori«c; . or lo 

the "old" program, Va >.ts ability < > m.:rL^ai:fe citt^dont learning. 

However,, two oi the Positive ObJ^ic^ivey d:,> lend chmaelvei* t j prograjii 
evaluation 'joniixderations . Thei;y ^re;; 

* Studentis involved in thi? LRC apptoaoh cla&fc*es 
will have better iielf Imag&jf^ vxf theinitelves as 
stodents and more favorable a(:?;itudei;j toward 
school than studeni:s Jn para I J til traditional 
claeseB . (affective dot&aln) 

* Students In the LRC approach coursjes will real- 
ize greater learning gains than students enrolled 

in parallel but traditional classes., (cognitive doinain) 

Thus^ the evaluation design offered here will be dealing with both 
program and process variables, hue will have a major emphasis on program 
considerations , 

Overall evaluation design , in this proposed evaluation two levels are 
evidento One is concerned with evaluating the individual courses and the 
other looks at the eifeotiveness of the program school*-wide. In each level 
of the evaluation<, both program and proce£(S variables are considered. No 
Instrtnnents, measurement techniques , or statistical methods are advanced as 
absolutes at this time, feeling that these will be determined by the evaluator 
actually conducting the final outcome evaluation » 
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IV. EVALUATION DESIGN 

Two schematics are presented as detailed explanations of the out-- 
come evaluation process, one is for use with individual courses and the 
other for evaluating overall program impact. 

Individual program evaluation , The Individual instructional program 
evaluation is perhaps the most critical from the educatiotist point of 
view as it will serve to determine If the "new" programs were more or Josa 
efficient ae instructional programs than the traditional ones. 

This evaluation phase will seek to answer such specific questions as: 

(1) Were the LRC classes better » and did they 
cost more or less? 

(2) Were the JRC classes the same and did they 
cost more or less? 

(3) Were the LRC classes worse and did they cost 
more or less? 

To do this two major areas are considered ^ The first is that of 
Learning Costs » and the other Learning Gains of investigations in these 
areas will make use of student/unit cost analysis, mastery exams > stan- 
dardized exams y and will reveal answers to the following questions: 

(1) How much did each program cost? 

(2) Which program was most effective? 

(3) What was outcome relationship between L&CC 
and nation? 

It could seem that mich of this evaluation will be concerning Itself 
with process variables based on costs and program variables based on 
standardised exams (if available) . This Is not so and it should be stated 
that ea^hasls on reivlewing the evaluation results should be placed on 
mastery of learning criteria » 

To do tills, it is suggested that Instructors for both the UlC ac^ 
the traditional parallel courses construct behavioral objectives. These 
objectives should be flagged and placed In a coosKm pool £rom which test 
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Individual program cvau^iation (.jonliinusd) 

items will he hV:Vx.t Exarny -jhoald have r mixture ot brlh LRC and tradrt Aonu j 

items and be adisinlstered in parallel form t«> both i<.And3t ol* classes^ 

if 

Thu6^ results from thr .we tents will indlv.ace ho^•7 well ctudenia 'Id on 
material associated with chia j.r claaw a^3 veil a^- t;hc counter lorn. Thit* will 
provide not only a mearturt -A how well er h uuvrsc did in meeting j j:s ov«^rall 
objectives^ but aJ.ao it wrli give a rnean-^ of Conning a legitimate intur - ia^s 
comparison » 

Overall prof^ram evaluati po y This phase of the evaluation Is tnoire c:o;>ct^rT:t^d 
with process variables and will seek to measure the g«»al attainment expreatiecf 
by faculty and adminiiitratlon In Lts earlier ''needs atiwescment." InveriLrgaiive 
procedures as shown in the 3i;hciiiatic, revolve about administration. studentM.^ 
and faculty. Again questions will be ordered co discover cost-effectiw. re- 
lationships o student attaini&ent as to skills, attitudes and knowledge, a:-, well 
as faculty sati8fai*tlon wi.th the progranit. 

Investigative techniques will hinge on the analysis of cost data and budgat 
files for LRC^ traditional and other cour&es. test and attltudinal toeasure out- 
comes for students resulting from the individual course evaluations ^ and measured 
faculty attltudinal changes along wi.th general monitoring of various process 
variables, 
V, SlRtfARY AND CX)NCII7SI0NS 

SUMMARY ^ What has been presented here is a guideline to a tvo<-stage out- 
come evaluatlCTa for individual courses within a total Learning Resources Center 
project* Program and process variables are both taken into consideration with 
emphasis on evaluating change in student behavior o The use of Behavioral ob-* 
jectlves has been cited as the best possible means of detendng the difference 
betveen ^treatment groups > and a scheme for so using them has been advanced . Two 
f lovcbarts are provided to serve as guides in conducting the evaluation tor thesf* 
^^9^. staff meters who may follow this reports 
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uoKnary snd conclUBi.jn& (continued) 

COWCUjSiON S > The moBt Inportant cont;lderat. tou h^re Is a philo jophy of 
evalttatiori; rather than a receipe ol* ho\^ to d * onu. Sufficieui: room e^cii^ts for 
the actual c*valuator to exp'.cenQ his Individual icy in treot and maayurero.'^nc de- 
sign and construction^ and hence Tnan.»€e<^l I'a ;rv/a peraonal /:y in thr- final ef- 
fort . 
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